Papers with long-tailed learning
Transformer Language Models Handle Word Frequency in Prediction Head (2023.findings-acl)
Copied to clipboard
| Challenge: | Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, its characteristics have been overlooked in previous analyses. |
| Approach: | They examine the inner workings of the prediction head, specifically the bias parameters, and quantify the effect of controlling their frequency biases on text generation. |
| Outcome: | The prediction head is a crucial component of the Transformer language models. |